Overview

Before reading this guide, please check out the admin guide to better understand how Tavern is deployed and managed. If you would like to help contribute to Tavern, please take a look at our open issues.

GraphQL API

Creating a New Model

  1. Initialize the schema cd tavern && go run entgo.io/ent/cmd/ent init <NAME>
  2. Update the generated file in tavern/internal/ent/schema/<NAME>.go
  3. Ensure you include a func (<NAME>) Annotations() []schema.Annotation method which returns a entgql.QueryField() annotation to tell entgo to generate a GraphQL root query for this model (if you’d like it to be queryable from the root query)
  4. Update tavern/internal/graphql/gqlgen.yml to include the ent types in the autobind: section (e.g.- github.com/spellshift/realm/tavern/internal/ent/<NAME>)
  5. Optionally update the models: section of tavern/internal/graphql/gqlgen.yml to bind any GraphQL enum types to their respective entgo generated types (e.g. github.com/spellshift/realm/tavern/internal/ent/<NAME>.<ENUM_FIELD>)
  6. Run go generate ./tavern/... from the project root
  7. If you added an annotation for a root query field (see above), you will notice auto-generated the query.resolvers.go file has been updated with new methods to query your model (e.g. func (r *queryResolver) <NAME>s ...)
    • This must be implemented (e.g. return r.client.<NAME>.Query().All(ctx) where NAME is the name of your model)

Adding Mutations

  1. Update the mutation.graphql schema file to include your new mutation and please include it in the section for the model it’s mutating if applicable (e.g. createUser should be defined near all the related User mutations)
    • Note: Input types such as Create<NAME>Input or Update<NAME>Input will already be generated if you added the appropriate annotations to your ent schema. If you require custom input mutations (e.g. ClaimTasksInput) then add them to the inputs.graphql file (Golang code will be generated in tavern/internal/graphql/models e.g. models.ClaimTasksInput).
  2. Run go generate ./...
  3. Implement generated the generated mutation resolver method in tavern/internal/graphql/mutation.resolvers.go
    • Depending on the mutation you’re trying to implement, a one liner such as return r.client.<NAME>.Create().SetInput(input).Save(ctx) might be sufficient
  4. Please write a unit test for your new mutation by defining YAML test cases in a new testdata/mutations subdirectory with your mutations name (e.g. tavern/internal/graphql/testdata/mutations/mymutation/SomeTest.yml)

Code Generation Reference

  • After making a change, remember to run go generate ./... from the project root.
  • tavern/internal/ent/schema is a directory which defines our graph using database models (ents) and the relations between them
  • tavern/generate.go is responsible for generating ents defined by the ent schema as well as updating the GraphQL schema and generating related code
  • tavern/internal/ent/entc.go is responsible for generating code for the entgo <-> 99designs/gqlgen GraphQL integration
  • tavern/internal/graphql/schema/mutation.graphql defines all mutations supported by our API
  • tavern/internal/graphql/schema/query.graphql is a GraphQL schema automatically generated by ent, providing useful types derived from our ent schemas as well as root-level queries defined by entgo annotations
  • tavern/internal/graphql/schema/scalars.graphql defines scalar GraphQL types that can be used to help with Go bindings (See gqlgen docs for more info)
  • tavern/internal/graphql/schema/inputs.graphql defines custom GraphQL inputs that can be used with your mutations (e.g. outside of the default auto-generated CRUD inputs)

YAML Test Reference (GraphQL)

Field Description Required
state SQL queries that define the initial db state before the query is run. no
requestor Holds information about the authenticated context making the query. no
requestor.beacon_token Session token corresponding to the user for authentication. You may create a user with a predetermined session token using the state field. no
query GraphQL query or mutation to be executed yes
variables A map of variables that will be passed with your GraphQL Query to the server no
expected A map that defines the expected response that the server should return no
expected_error An expected message that should be included in the query when it fails no

Resources

GRPC API

Tavern also supports a gRPC API for agents to claim tasks and report execution output. This API is defined by our c2.proto spec and is still under active development.

Downloading Files

You may download files from Tavern utilizing the DownloadFile gRPC method. This method streams responses, each of which will contain a chunk of the desired file. We rely on the ordering guarantees provided by gRPC to ensure the file is assembled correctly. This API also sets two headers to ensure the integrity of files:

  • sha3-256-checksum: Set to the SHA3 hash of the entire file.
  • file-size: Set to the number of bytes contained by the file.

Performance Profiling

Tavern supports built in performance monitoring and debugging via the Golang pprof tool developed by Google. To run tavern with profiling enabled, ensure the ENABLE_PPROF=1 environment variable is set.

Install Graphviz

Ensure you have an updated version of Graphviz installed for visualizing profile outputs.

apt install -y graphviz

Collect a Profile

  1. Start Tavern with profiling enabled: ENABLE_PPROF=1 go run ./tavern.
  2. Collect a Profile in desired format (e.g. png): go tool pprof -png -seconds=10 http://127.0.0.1:80/debug/pprof/allocs?seconds=10 > .pprof/allocs.png a. Replace “allocs” with the name of the profile to collect. b. Replace the value of seconds with the amount of time you need to reproduce performance issues. c. Read more about the available profiling URL parameters here. d. go tool pprof does not need to run on the same host as Tavern, just ensure you provide the correct HTTP url in the command. Note that Graphviz must be installed on the system you’re running pprof from.
  3. Reproduce any interactions with Tavern that you’d like to collect profiling information for.
  4. A graph visualization of the requested performance profile should now be saved locally, take a look and see what’s going on 🕵️.

Agent Development

Tavern provides an HTTP(s) gRPC API that agents may use directly to claim tasks and submit execution results. This is the standard request flow, and is supported as a core function of realm. For more information, please consult our API Specification.

If you wish to develop an agent using a different transport method (e.g. DNS), your development will need to include a C2. The role of the C2 is to handle agent communication, and translate the chosen transport method into HTTP(s) requests to Tavern’s gRPC API. We recommend reusing the existing protobuf definitions for simplicity and forward compatability. This enables developers to use any transport mechanism with Tavern. If you plan to build a C2 for a common protocol for use with Tavern, consider submitting a PR.

Agent Loop Lifecycle

  1. Claim Tasks
  2. Execute Tasks (happens in parallel and may not finish within one loop)
  3. Report available output from Task execution
  4. Sleep for an interval and repeat